Reactivate the agentic eval against the rebuilt CLI - #183
Merged
Conversation
The eval workflow and its prompt still spoke the pre-rebuild language — the Snakemake shim, `lc run <output_id> --universe`, ok/stale/missing states, the Containerfile + requirements.txt environment. Bring both up to what the CLI is now: - prompt.md: `lc materialize` is the build verb and `lc run` the probe; states are current/behind/stale and `lc materialize --check` is the gate; recipes import from the project's own locked environment (`uv add`, never `uv pip`) under the sandbox; a run refuses a dirty tree, so the agent commits its edits with plain git; `results/` is the engine's to write, and a foreign write is detected. The seed spec is now stated to be deliberately incomplete — completing the recipe templates and dependency contracts is part of the task. - eval.yml: the check step gates on `lc materialize --check --json` (plus a non-empty `.current`) instead of a jq shape no verb emits; setup seeds numpy/scipy/matplotlib into the project with `uv add` and commits the seed so the agent starts from a clean tree; the repo-venv `uv pip install` is dropped (recipes never saw it); the dormant PR plumbing gets its trigger back as a `run-eval` label; the artifact exclusions swap `.snakemake/` for `.lightcone/`. - The seed astra.yaml drops its `container:` key — the directive is ignored by decision, and no Containerfile exists to point at. - trace-analysis-prompt.md names the sandbox instead of the container environment. Verified locally: the seed validates against astra-tools 0.2.16, and the setup + check plumbing runs end-to-end against today's `lc` (the gate correctly fails with all three outputs planned on the unbuilt project). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
The setup step no longer pre-seeds numpy/scipy/matplotlib — declaring what the recipes import is part of what the eval measures. In exchange the prompt spells out the uv workflow: the project starts with no dependencies, `uv add` is the one way packages reach a recipe (it edits pyproject.toml, re-locks, and syncs in one step, and the result is committed like any other edit), and lock-bypassing installs reach nothing a recipe sees. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
Contributor
✅ Eval
lc statusConfusion & pain points (Claude analysis)Confusion & pain points
Full trace: |
The engine is the host's uv tool, so the eval now installs it that way: `uv tool install` of the checkout — the commit under test, stamped into the version by hatch-vcs — links lc and, through the mirrored entry points, the git-annex executables, exercising the exact mechanism a user's `uv tool install lightcone-cli` relies on. astra's executable belongs to astra-tools and becomes its own tool install, pinned to the branch's dependency version. The repo venv and its activation are gone with `uv sync`; nothing sets VIRTUAL_ENV anymore, which also removes the benign environment-mismatch warning uv printed at every project-scoped `uv add`. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
After every output is materialized, the agent is asked to declare an SPDX license in pyproject.toml and materialize once more — publication intent derived exactly as layer 8 designed it, and the crate converges on a run that remakes nothing. The check step gates on the crate file existing, so the RO-Crate path is exercised by every eval run instead of warning its absence into the report. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx
EiffL
added a commit
that referenced
this pull request
Aug 21, 2026
Widens #183's label gate: the eval runs on every non-draft PR, with `ready_for_review` beside the default types so flipping a draft to ready triggers the run the draft guard skips. The per-PR concurrency group already cancels superseded runs, so a push train costs one eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016CRERrB5zWdVXD6uPv2BRa
EiffL
added a commit
that referenced
this pull request
Aug 21, 2026
…ibility (#190) This merges the clean rebuild of lightcone-cli onto main — the full re-architecture from the Snakemake shim to the model where **the locked environment is the execution environment**. The pre-rebuild codebase was deliberately stripped and functionality re-added one layer at a time, each layer landing with its own tests, dependencies, and (since the rewrite) docs. 21 PRs, all individually reviewed and merged into `clean_rebuild`; this PR is the promotion. ## What lightcone-cli is now A project is `pyproject.toml` + `uv.lock` + `astra.yaml` — uv is the only environment substrate, and mode is derived, never configured: direct by default (recipes in the project's `.venv` under Landlock/Seatbelt), containerized when `[tool.lightcone.image]` is declared (recipes in a content-addressed image archived in the repository itself). Five verbs: `init`, `materialize`, `status`, `run`, `build`. - **Identity**: `definition_version` (what an output *is*) and `env_version` (what it ran under) are separate on purpose — a spec contradiction is `stale` and remade; an environment move is `behind`, reported and left alone. `--refresh` widens by exactly that one state. - **Storage**: the DataLad model — git carries history, git-annex carries bytes, behind ordinary `git add`/`git commit`. Every output is committed with a manifest and a run record `datalad rerun` can replay; runs start and end on a clean tree; foreign writes are detected by history and classify stale. - **Execution**: one exec boundary (`Policy` → pure argv rewrite → honest attestation) shared by recipes and the `lc run` probe; a recipe writes its own output directory and nothing else; denials carry copy-pasteable remedies. Network is uncontrolled on every mechanism, symmetrically attested. - **Venues**: detected, never configured — a SLURM allocation is spanned with one worker per node; known centers get a login guard whose refusal prints the center's own `salloc`/`sbatch` spellings; podman-hpc makes multi-node containerized runs real. - **Publication**: declaring `[project].license` turns on the RO-Crate view — `ro-crate-metadata.json` converged by materialize, validated against Provenance Run Crate 0.5 in CI. Deposit is `git archive`. - **The engine is the host's uv tool, never a project dependency** — a project's lock carries only what the analysis imports, and an engine upgrade rebuilds nothing. ## The layers, as merged | PR | Layer | |---|---| | #173 | Layer 5: the exec boundary and `lc run` (landed first — the smallest honest consumer of the boundary) | | #175 | Layers 2 + 4: identity, provenance, `lc materialize` | | #178 | The engine is the host's uv tool (deletes layer 3 by decision) | | #180 | Layer 6: the container hatch | | #181 | Layer 7: venues — SLURM in-allocation, podman-hpc | | #182 | Layer 8: the publication view | | #184 | Hardening: the recorded residues closed (UV_* scrub, write-scope narrowing, symmetric network attestation, machine-level uv.toml advisory) | | #183 | The agentic eval reactivated: `uv tool install` of the branch under test, empty-environment task, RO-Crate gate — three green runs | | #185–#189 | The documentation rewritten end to end: README, user guide, CLI reference, developer corner, governance — every command block executed before shipping, every refusal quoted from a real run | ## Verification - The suite (tests, lint, mypy strict) is green on `clean_rebuild`, including the three gated suites CI requires: sandbox enforcement (Landlock on Linux, Seatbelt on macOS), container smoke (real build → archive → bytes-free-clone `datalad rerun`), and crate smoke (the official validator, REQUIRED clean). - The agentic eval passes end-to-end on this branch: an agent builds a supernova-cosmology analysis from the spec, manages the environment with `uv add`, materializes through the real engine, and publishes a crate — $0.70–$1.22 per run. - `zensical build` renders the rewritten docs with zero issues; the site deploys on the next release. ## Known deferred work (tracked, not blocking) - The one-time **Perlmutter spike** (layer 7's pending items are listed in CLAUDE.md) — HPC support ships honestly labeled "early days". - **hub/GKE + Cloud Build** venue and **apptainer/singularity** runtimes — design headroom recorded, deliberately not stubbed. - Multi-arch image archives; the design-doc drop and CLAUDE.md slimming when the rebuild formally closes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The eval workflow and its prompt were written for the pre-rebuild CLI — the Snakemake shim,
lc run <output_id> --universe,ok/stale/missingstates, the Containerfile +requirements.txtcontainer model. With layer 8 landed, this brings the eval back in line with whatlcactually is.evals/prompt.md(rewritten)lc materializeis the build verb;lc run <command>is the probe. States arecurrent/behind/stale,lc statusalways exits 0, andlc materialize --checkis the gate.uv addin the project root (what the denial messages recommend);uv pipreaches nothing a recipe sees.git add/git commit(the annex filter is transparent — nobody runs a git-annex command).results/is the engine's to write; a hand-placed file is now detected as a foreign write and remade — the old "fails verification" warning got teeth.{output},{inputs.<id>},{decisions.<id>}, and the matchinginputs:/decisions:lists) is part of the task. This makes the existing seed's shape an intentional test of spec literacy rather than a leftover.astra guide, astra-tools 0.2.16)..github/workflows/eval.yml.universes[].outputs[],"ok","alias") matches nothing any verb emits now. Replaced with the designed gate —lc materialize --check --json | jq -e '.up_to_date and ((.current | length) > 0)'— which exits nonzero while anything is planned or failed, plus the non-emptiness guard the old check had.behinddeliberately does not fail the gate.uv add, spelled out in the prompt) is part of what the eval measures; the repo-venvuv pip installis dropped (it never reached recipes). Setup commits the seed —lc initnever commits, and the firstlc materializewould otherwise be refused on a dirty tree before the agent did anything.pull_request: types: [labeled]gated on arun-evallabel, alongsideworkflow_dispatch..snakemake/**exclusion →.lightcone/**; keeping.gitviainclude-hidden-filesis now a feature — the run records and annexed results are the provenance.uv tool installof the checkout (the commit under test, stamped into the version by hatch-vcs) linkslcand the four git-annex executables through the mirrored entry points, exercising the exact mechanism a user install relies on; astra-tools is its own tool install, pinned to the branch dependency. No venv is activated, so the runner needs no system git-annex andVIRTUAL_ENVis never set.Seed + trace prompt
evals/tasks/snae/astra.yamldropscontainer: Containerfile— the directive is ignored by recorded decision, and no Containerfile exists to point at.trace-analysis-prompt.mdnames the sandbox instead of the container environment.Verified locally
lc:lc init→ overlay →astra universe generate -n baseline→uv add→ seed commit →lc status(three outputs, direct mode, landlock) → the--checkgate correctly failing with all three outputs planned on the unbuilt project.The agent step itself (Claude Code + astra plugin) is unchanged and needs a real dispatch run to shake out — suggest a
workflow_dispatchon this branch once merged, or label this PRrun-eval.🤖 Generated with Claude Code
https://claude.ai/code/session_01DJzmp2MUhwiNHR94cB91dx